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Abstract 

Soar  is  an  architecture  for  intelligence  that  integrates  teaming  into  all  of  its  problem-solving  behavior.  The  ’ 
learning  mechanism,  chunking,  hu  been  studied  experimentally  in  a  broad  range  of  and  situations.  This 
paper  summarizes  the  research  on  chunking  in  Soar,  covering  the  effects  ^  chunking  in  different  tasks, 
task-independent  applications  of  chunking  and  our  theoretical  analyses  of  effects  and  limits  of  chunking.  We 
discuss  what  and  when  Soar  has  been  able  to  learn  so  far.  The  results  demonstrate  that  the  variety  of  learning  in 
Soar  arises  Grom  variety  in  problem  solving,  rather  that  from  variety  in  architectural  mechanisms. 


1.  Introduction 

Soar  is  an  archiiecaire  that  is  to  be  capable  of  general  intelligence,  displaying  abilities  in  areas  such  as 
problem-solving,  planning,  diagnosis,  learning,  etc.  (Laird,  Newell  &  Rosenbloom,  ??).  The  architecuire 
contains  a  single  learning  mechanism,  called  chunking,  which  saves  the  results  of  processing  for  application  to 
future  similar  situations  (Laird.  RosenUoom  A  Newell,  1984,  Laird,  Rosenbloom  A  Newell,  1986a).  Chunking 
was  first  incorporaied  into  Soar  in  early  1984  as  a  particular  learning  mechanism  we  had  been  exploring  in  the 
context  of  human  practice.’.  Almost  munediaidy  thereafter  chunking  proved  itself  capable  of  several  different 
types  of  learning  (Laird,  Rosenbloom  A  NewdL  1984.  Rasenbloom,  Laird,  McDermott,  Newell  A  Ociuch, 
1985).  At  first  tentatively,  but  than  widi  incteasinf  commitment,  we  adopted  the  hypothesis  that  chunking  was 
a  sufficient  mechanism  for  aU  learning. 

We  have  not  —gf««  in  a  delfoerate  attempt  to  test  this  hypothesis  in  any  definitive  way.  There  currently 
exists  no  taxonomy  of  learning  dtnations  or  processes  that  is  well-founded  enough  to  support  such  an  endeavor, 
although  partial  classificadons  abound,  such  as  learning  by  experience,  learning  by  example,  and  learning  by 
instruction.  Instead  we  have  proceeded  to  explore  die  operation  of  Soar  in  many  different  directions,  observing 
the  way  learning  entered  imo  the  various  tasks  and  mecfaaniams  studied. 

The  variety  of  learning  exhibited  by  Soar  seems  extensive  enough  by  now  to  make  a  listing  of  all  the  various 
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examples  worthwhile.  That  is  the  objective  of  this  paper.  First,  we  very  briefly  describe  the  s&ucture  of  Soar. 
Then  we  list  the  diffieieiu  types  of  teaming,  using  the  outline  of  Figiue  1-1.  By  necessity  this  listing  treats  each 
of  die  eotamides  of  in  its  own  lenns.  Furthermore,  it  isolates  them  from  the  larger  project  context  in  which  they 
have  occurred  (and  indeed  many  pots  of  the  total  Soar  project  receive  no  mention  in  this  list).  We  then 
ledescribe  this  pool  of  instances  from  a  more  systematic  standpoint  Even  here  the  categories  we  use  are 
essentially  ad  hoc,  reflecting  more  our  attempt  to  understand  the  diversity  of  what  chunking  has  done  in  Soar 
than  any  a  prioti  theoreticai  view  of  learning. 

Effects  of  chunldaf  in  different  tanict 
Puzzles  and  toy  probtems 
Syllogisms 

BalaiiM  beam  (children’s  development) 

Seibei’s  task  (stimulus-response) 

Rl-Soar  (VAX  computer  configuration) 

Cypress-Soar  (algorUhm  design) 

Task-tadependent  applications  of  chunking 
Task  acquisition 
Data  chunking 
Macro-operators 
(Constraint  compilation 
Learning  from  outside  guidance 
Abstraction  planning 
Explanation-based  generalization 
Theoretical  analysis  of  chunking 
Sources  of  overgeneralization  in  Soar 
Meta-levels  in  Soar 


Figure  1-1:  Summary  of  Soar  research  relevant  to  teaming 


2.  The  Soar  architecture 

The  following  is  a  brief  description  of  how  Soar  solves  problems  and  learns  during  the  process.  More 
complete  descriptions  of  Soar  are  available  in  (Laird,  1986)  and  (Laird,  Newell  St  Rosenbiooffl,  ??). 

Soar  is  based  on  the  hypothesis  that  all  goal-erected  cognitive  activity  can  be  represented  as  search  in  a 
problem  space.  A  problem  space  is  defined  by  a  set  of  states,  and  a  set  of  operators  to  move  from  state  to  state. 
Given  a  task,  reptesenied  as  a  set  of  problem  spaces.  Soar  uses  knowledge  to  select  problem  spaces,  states,  and 
opeiaiars  to  move  lowaidi  a  desired  state.  This  knowtedge  is  represented  as  productaons,  which  ate 
syntactically  similar  to  (3psS  (Foigy,  1981)  rules.  When  the  knowledge  diteedy  avaiWile  in  a  given  context  is 
insufficient  to  determine  the  next  thing  to  do  immediaiely.  Soar  sets  up  new  contexts  in  which  it  attempts  to 
obtain  the  necesnry  knowledge. 

Soar  operates  in  terms  of  deeuion  cycUs,  The  decisions  relate  to  progressing  towards  a  desired  state: 
selecting  problem  spaces  to  work  in,  sates  to  proceed  from,  and  operators  to  apply.  Several  productions  may 
Are  in  accmiHilating  the  knowledge  to  make  a  given  dedaioiL 

The  sioiations  in  which  Soar  is  unable  to  make  progtess  on  some  goal  because  of  inadequate  directly 
available  knowledge  are  called  impasses.  The  types  (rf  impasses  that  may  arise  in  Soar  systems  are  determined 
by  the  architecture.  For  mtample,  a  common  iropaaae,  operator-tie,  occurs  when  several  operators  are  proposed 
as  acceptable  for  application  to  a  given  state,  and  there  is  insufGciem  knowtedge  to  choose  between  them.  Soar 
exclusively  sets  up  subgoals  to  resttive  such  impasses.  In  order  to  work  on  a  subgoal.  Soar  applies  all  its 
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problem  solving  machinery  just  as  it  did  for  the  higher-level  goals  (selecting  a  problem  space  to  work  on,  etc.). 
The  subgoal  to  resolve  an  operatar-de  impasse  is  satisfied  when  the  system  acquires  knowledge  indkadng  that 
one  of  the  operwots  initially  causing  the  de  is  now  prefer^e  to  all  other  candidates. 

When  Soar  finidies  working  on  a  sidtgoal,  it  can  learn  from  its  experience  by  building  produedons  called 
chtmks  for  use  in  future  problem  solving.  The  conditions  of  a  chunk  are  the  features  of  the  pre-impasse 
situation  that  were  tested  during  work  on  the  subgoal  to  resolve  the  impasse.  The  actions  of  a  chunk  are  the 
attributes  added  to  the  situation  that  are  connected  to  die  embedding  supercontexu  some  of  these  attributes 
teiminated  the  impasse.  At  first  glance  one  would  expea  chunking  to  yield  nothing  mote  than  rote  learning,  but 
generalizadon  does  occur  because  chunks  test  only  relevant  attributes  of  the  problem-solving  context  (Laird, 
RosenUoom  &  Newell,  1984). 

3.  Soar  research  relevant  to  learning 

This  secdon  describes  the  problem,  approach,  and  results  obtained  for  each  topic  listed  in  Figure  1-1. 

3.1.  Efflects  of  chunking  in  different  tasks 

Puzzles  and  toy  problems:  The  first  demonstradons  of  the  power  of  chunking  as  a  learning  mechanism  were 
obtained  with  tasks  such  as  solving  the  eight-puzzle  and  playing  dc-tac-toe  (Laird,  Rosenbloom  k  Newell, 
1984).  These  results  include  within-trial  transfer,  across-trial  transfer,  across-task  transfer,  and  strategy 
acquisition.  One  interesting  feature  is  that  Soar  was  able  to  leam  search  control  knowledge  that  generalized 
across  symmetric  board  positions,  even  though  the  represintation  of  the  tasks  did  not  include  explicit 
knowledge  of  symmetry.  Since  then.  Soar  systems  have  been  implemented  to  solve  many  of  the  standard  AI 
toy  problems,  and  learning  behaviOT  has  been  studied  in  nearly  all  of  these  systems. 

Syllogisnu:  Mental  models  have  been  proposed  as  an  alternative  to  deduction  as  an  account  fw  human 
reasoning  performance.  Our  work  on  building  mental  models  within  Soar  has  centered  on  the  task  of  solving 
syllogisms:  given  two  premises,  what  conclusions  necessarily  follow?  For  example,  given  that  some  B  are  A, 
and  no  C  are  B,  is  it  necessarily  true  that  some  A  are  not  C?  Psychological  studies  of  people  perfonning  this 
task  show  that  syllogisms  vary  widely  in  difliculty.  People  don’t  necessarily  make  logically  valid  inferences; 
indeed,  in  some  syllogisms,  they  nearly  always  make  mistakes.  Johnson-Laird  (Johnson-Laird,  1983)  has 
developed  a  theory  that  says  people  solve  them  by  building  menuti  models  of  possible  worlds  that  could  be 
described  by  the  premises,  and  checking  them  lo  see  if  the  conclusion  holds.  Ife  claims  the  syllogisms  that  are 
the  hardest  for  people  to  solve  are  those  for  which  it  is  necessary  to  construct  the  most  models.  We  are  in  the 
midst  of  an  effort  to  map  Johnson-Laird’s  theories  into  Soar.  Chunking  plays  a  strong  role  in  this  mapping,  in 
terms  of  teaming  to  draw  valid  oonclusions.  Once  Soar  teams  an  chunk  stating  a  coiKlusion  is  valid  in  one 
model,  that  production  will  fire  in  any  other  model  where  that  conclusion  is  valid. 

Balance  beam:  Mach  of  the  teaming  research  in  Soar  centers  on  the  kind  of  teaming  that  humans  display  in 
the  short-  to  medinin-tenn,  which  raiaes  an  mieresting  question:  Can  long  term  teaming,  i.e.  the  developmental 
bdwvior  that  chikben  exhibit  over  several  years,  also  be  modeled  by  chunking,  or  are  diffeiem  architectural 
mechanisms  required?  In  order  to  discover  what  rote  Soar  may  |day  in  explaining  the  developmem  of  cognitive 
abilities,  we  are  building  models  of  probtem-solving  fora  task  studied  by  Siegter  (Siegter,  1978).  Given  a  beam 
balanced  on  a  fulctum  at  the  center,  the  problem  is  to  determine  which  side  of  the  beam  will  go  down  when 
certain  weighu  are  placed  at  varying  disiances  on  either  side.  Children’s  abilities  to  solve  this  problem  go 
through  (bur  fitiriy  well-defined  stages  in  the  rules  used  for  making  predictioas.  Initially,  they  base  their 
predictioas  only  on  the  weights.  Then  they  consider  the  diatances  in  order  to  break  ties  on  the  weights,  then 
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they  break  ties  by  combining  the  weights  and  distances  in  some  fashion,  and  finally  they  use  the  weights  and 
distancea  to  compate  the  comet  torques.  We  are  investigating  the  hypothesis  that  chunking  can  be  used  to 
model  this  gradual  increase  in  sophistication  in  solving  this  task.  The  Soar  model  of  the  balance  beam  task 
maps  development  onto  a  shift  in  the  problem  spaces  used  by  the  subject  Initially  Soar  has  a  set  of  attributes 
about  the  domain  (such  as  weight  and  distance),  but  knows  neither  exactly  which  attributes  are  relevant  nor 
how  to  connect  the  attributes  to  the  operators  that  predict  which  side  of  the  balance  beam  will  go  down.  Soar 
makes  predictions,  some  of  which  are  shown  to  be  wrong  by  observation,  and  uses  the  results  as  feedback  to 
create  new  problem  spaces  that  make  better  predictions. 

Seibd:  As  part  of  an  investigation  into  the  eftects  of  practice,  we  are  examining  a  reaction-time  task 
investigated  by  Seibel  (Seibel,  1963).  The  task  is  a  psychological  experimem  in  which  the  subject  faces  a 
display  of  ten  lights,  and  rests  his  or  her  fittgers  on  a  set  of  ten  buttons;  during  each  trial,  some  of  the  lights  will 
go  on,  and  the  subject  is  to  press  the  buttons  cmiesponding  to  the  lights  that  are  on  as  quickly  as  possible.  The 
reaction  time  data  for  this  task  obey  a  power  law,  and  earlier  work  showed  that  the  power  law  could  also  be 
obtained  from  a  computer  simulation  of  the  process  based  on  chunking  (Rosenbloom  &  Newell,  1983).  Soar’s 
chunking  mechanism  has  a  mode  in  which  it  only  builds  chunks  fw  terminal  subgoals,  which  we  call  bottom-up 
cluuikuig.  and  which  can  account  for  the  effects  of  practice  under  certain  assumptions.  In  recent  work,  we  have 
leplicaied  the  power  law  effect  for  the  Seibel  task  in  Soar,  under  the  assumption  that  reaction  times  correspond 
to  number  of  decision  cycles  to  solution. 

Rl<So*r  Rl-Soar  was  the  first  demonstration  that  Soar  can  combine  general  problem-solving  methods  with 
large  amounts  of  domain  knowledge  to  solve  complex  problems  (Roaenbloom,  Laird,  McDermott,  Newell  &. 
Oiciuch,  1983).  The  system  had  about  25%  of  the  functionality  of  Rl,  an  expert  system  for  configuring  Vaxes. 
Because  the  chunking  mechanism  caches  subgoal  results  during  knowledge-intensive  as  well  as  knowledge-lean 
problem-solving,  we  were  :d>le  to  perform  several  experiments  on  the  effects  of  learning  on  performance  for  the 
Rl-Soar  ask.  The  knowledge  gained  by  chunking  was  used  in  Rl-Soar  for  search  control  and  operator 
implementation.  Chunking  decreases  the  amount  of  effort  required  in  problem-solving  even  when  the  system 
already  has  some  search  control  knowledge  (a  factor  of  3.6  for  the  version  without  search  control,  a  factor  of  2.2 
for  partial  search  controL  and  a  fKtor  of  1.7  for  full  search  control).  We  refer  to  this  reduction  as  within-trial 
tranter,  because  the  sysam  acquires  knowledge  early  on  that  it  may  apply  to  solve  later  subproblems  within 
the  same  trial  As  would  be  expected  born  an  expeiience-leamer,  knowledge  transferred  across  trials  was  also 
substantial,  yielding  a  solution  length  reduction  oX  between  a  factor  of  20  and  200  in  subsequent  runs  of 
Rl-Soar  on  the  same  ask.  There  turoed  out  to  be  little  effect  from  transfer  across  different  tasks. 

Cyprsas  Sonr:  Another  knowledgB-inieaaive  task  to  which  we  have  applied  Soar  is  algorithm  design  (Steier, 
1987).  Again,  we  took  the  approach  of  reimpleroenting  a  previously  built  system;  in  this  case,  Doug  Smith's 
semi-automatic  algorithm  designer.  Cypress  (Stitith,  IMS).  We  set  about  replicating  the  performance  of 
Cypress  in  the  design  of  three  sorting  algorithms:  insertion-sort,  quicksort,  and  mergesoit  Cypress  makes 
design  choioes  for  these  aigorithins  by  msontiating  a  functional  programming  temphue  for  divide-and-conquer 
algorithms,  and  pwTfrinf  conmahua  resulting  fiom  these  instantiations  by  a  special  method  of  deduction 
known  as  antecedent  derivation.  We  focused  only  on  the  methods  for  making  the  design  choices  in  Cypress- 
Soar,  and  chose  not  to  reimplemem  the  antecedent  derivation  engine.  As  in  Rl-Soar,  Cypress-Soar  was  able  to 
successfully  perform  ia  tads  with  varying  amooms  of  search  conuol  knowledge.  Uring  chunking,  Cypress- 
Soar  acquires  both  search  comrol  and  operator  hnpiememation  knowledge.  The  efCtct  of  within-trial  transfer  in 
one  ran  with  miniinal  search  control  was  to  reduce  the  number  of  decision  cycles  by  almost  70%.  Across-uial 
aansfier  on  design  of  the  same  algorithm  gave  reductions  of  90-97%.  The  operator  implemenotion  chunks 
aansferrad  across  derigns  of  tUftaent  algorithms,  reducing  the  number  of  decision  cycles  for  subsequent 
designs  by  as  much  as  20%.  We  expect  the  20%  across-osk  uansfer  figure  would  have  been  higher  had  we 


LearaiBg  in  Soar:  1987 


5 


implemented  a  Soar-based  deductton  engine. 

3^  Task'indepeiulent  appUcntions  of  chunking 

Task  acqnititloa:  Currently,  the  standard  way  for  Soar  to  acquire  new  tasks  is  to  a  "Soarwaie  engineer"  to 
analyze  the  task  and  write  a  set  of  productions  implementing  the  problem  spaces  necessary  for  performing  the 
ta-ok.  Since  Soar  is  intended  to  function  autonomously,  we  need  to  define  mechanisms  for  Soar  to  acquire  new 
task  spaces  on  its  own.  Our  intended  approach  used  a  problem  space  for  building  problem  spaces:  one  that 
would  compile  task  descriptions  directly  into  productions  through  deliberate  processing  (Le..  not  through 
chunking).  We  have  now  moved  to  an  approach  that  uses  of  the  combined  learning  and  problem  solving 
abilities  of  Soar.  Task  acquisition  is  implemented  in  a  two-phase  process.  The  first  phase  is  communication  in 
which  Soar  comprehends  the  input  and  store  it  internally.  The  second  phase  is  interpretation  in  which  Soar 
intetpretively  executes  the  task  description,  and  builds  chunks  to  directly  implement  the  task.  In  the 
communication  phase,  the  task  descriptions  are  input  in  pseudo-natural  language  in  terms  of  operators,  desired, 
initial,  and  illegal  states.  These  descriptions  are  understood  by  a  series  of  word-at-a-time  cmnprehension 
operaton,  in  a  method  based  on  Steven  Small’s  Word  Expert  Parser  (SmallSO,  1980).  These  iterators  either 
place  intomation  about  expectations  and  meanings  of  words  into  the  state  or  package  a  set  of  input  utons  into 
higher  level  units.  The  next  phase  evokes  a  problem  ^race  for  interpretation  in  the  event  of  impasses  resulting 
firom  attempts  ur  execute  the  task.  After  resolving  the  interpretation  impasses.  Soar  will  build  task- 
implementation  chunks,  so  that  the  interpreter  will  not  have  to  be  evoked  on  subsequem  task  trials.  These 
chunks  turn  out  to  be  nearly  identical  to  the  ones  that  a  programmer  would  have  written  by  hand  to  direcdy 
implement  the  task.  This  two-phase  process  has  now  enabled  Soar  to  acquire  the  eight-puzzk  and  missionaries- 
and-catmibals  task  spaces. 

Data  chunking:  The  evidence  accumulated  to  date  cleariy  demonstrates  that  chunking  can  speed  up  Soar's 
performance.  However,  until  recently  there  has  been  no  evidence  that  chunking  could  also  be  used  to  acquire 
new  factual  knowledge,  and  doubts  have  been  raised  as  to  whether  it  is  possible  to  such  an  experience-based 
learning  mechanism  to  do  so.  Our  work  on  data  chunking  is  an  attempt  to  demonstrate  how  chunking  can  be 
used  to  acquire  new  knowledge.  In  the  initial  phase  we  are  focusing  on  two  simple  memory  tasks:  learning  to 
recognize,  and  teaming  to  recall  new  objects  (Rosenbloom,  Laird  &,  Newell,  1987).  Recognition  involves 
determining  whether  the  system  has  ever  seen  the  object  before.  Recall  involves  the  ability  to  generate  a 
representation  of  the  new  object  on  demand.  For  recognition,  a  chunk  must  be  learned  which  contains  a 
representation  of  the  new  object  in  its  conditions,  while  for  recall,  the  object  must  be  represented  in  the  actions. 
Fbr  both  tasks,  the  general  approach  we  have  taken  is  to  understand  what  type  of  problem  solving  is  required  in 
order  to  achieve  the  appropriate  chunks.  For  recall,  the  more  interesting  of  the  two,  pertomance  is  based  on  a 
generation  problem  space  in  whkh  any  potential  object  can  be  generated.  This  is  a  constructive  space  in  which 
new  objects  are  built  up  out  of  known  pieces.  When  the  system  receives  an  object  that  it  should  leam  to  recaU, 
it  genetmes  a  repwaentation  of  it,  thus  learning  a  chunk  which  can  generate  it  in  the  funire.  At  recall  time. 
Soar’s  reflective  ability  is  used  to  distinguish  between  the  objects  that  it  has  learned  to  generate  and  those  that  it 
could  potentially  generate;  that  is,  when  all  of  the  chunks  have  fired,  an  impasse  occurs  which  the  system  uses 
as  a  signal  that  recall  should  be  terminamL  We  are  currently  working  on  two  more  advanced  memory  tasks: 
cued  recall  and  paired>associaie  recalL  In  cued  recall,  the  recall  of  an  item  is  conditioned  on  a  set  of 
discrifflinatiiv  cues  that  cm  be  used  as  the  basis  for  building  a  discrimination  network  for  retrieval.  This 
permits  retrieval  of  m  objea  by  partial  specifications  and  is  being  used  as  a  building  block  in  an 
implefflentatioo  dim  EPAM-like  system  (Feigenbaum  Si  Simon,  1984)  to  paired-associate  recall. 

Macnt-operatan:  Macro-operaiors  have  been  used  in  a  variety  of  systems  since  STRIPS  (Fikes,  Han  ft 
Nilsson.  1972)  to  improve  the  pertomance  of  problem-solving  systems.  A  macro-operator  is  a  sequmce  of 
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operators  that  can  be  treated  as  a  single  operator.  Korf  denunstitted  that  it  is  possible  to  define  a  table  of 
macro-operaton  showing  how  to  transfonn  all  legal  states  into  desired  states  if  the  problem  is  serially 
decomposable,  that  is,  the  sidigoals  of  the  problem  can  be  ordered  so  that  they  only  depend  on  preceding 
subgoals  (Korf,  1983).  It  is  both  passible  and  useful  to  build  macro  tables  for  tasks  such  as  the  eight-puzzle  and 
Rubik’s  cube.  Mduis-ends-analysis  and  hill-climbing  don’t  work  for  these  tasks,  because  the  problem  solver 
has  to  temporarily  undo  previously  solved  subgoals  in  order  to  make  progress.  We  have  replicated  Korf  s 
results  in  Soar  by  using  two  proMeir  spaces:  one  for  the  domain  operations  of  the  "conventional’’  task 
decomposition,  and  one  for  operators  coneqionding  to  setially-decomposable  goals  (Laird,  Rosenbloom  & 
Newell.  1986a.  Laird,  Newell  A  Rosenbloom.  ??).  The  sets  of  search  control  chunks  learned  in  the  process  of 
implementing  each  operator  in  the  second  space  correspond  to  macro-operators.  In  a  situation  identical  to 
Korf  s  macro-operator  learner,  in  which  there  is  no  transfer  between  macro-operators,  it  would  take  230 
productions  to  encode  the  entire  macro  table  (33  macro-operators)  this  way.  Htowever,  Soar  needs  only  61 
productions  to  encode  the  table  because  of  the  implicit  generalization  performed  by  chunking.  The  fine-grained 
encoding  of  the  macro-operators  as  sets  of  chunks  and  the  choice  of  an  appropriate  task  represenution  makes 
this  possible.  More  specifically,  the  generality  arises  because  the  operators  are  indifferent  to  the  dies  not  yet  in 
place,  and  the  chunks  are  invariant  with  respect  to  reflection,  rotadon,  iransladon,  and  position  within  the 
solution  path.  In  fact,  so  much  transfer  occurs  with  this  representation  that  Soar  can  learn  the  entire  macro  table 
for  the  eight-puzzle  in  only  three  trials  (given  the  correct  set  of  probtems).  Similar  results  were  obtained  for  the 
Towers  of  Hanoi  puzzle.  Soar  was  able  to  encode  the  entire  macro  table  (six  macro-opentots)  in  eight  chunks 
afler  a  single  trial 

Constraint  compilation:  Many  tasks  that  have  been  studied  in  artificial  intelligence  can  be  viewed  as 
constraint  satisfacdon  problems  (Steele,  1980).  Examples  are  cryptarithmedc,  eight-queens,  and  the  Waltz  line 
labeling  task  (Waltz,  1973).  If  one  views  the  satisfacdon  of  a  constraint  as  the  applicadon  of  an  operator,  then 
formulating  constraint  satisfacdon  as  problem  space  search  in  Soar  is  not  difficult  When  a  consistent 
assigiunent  of  values  to  all  variables  has  been  found,  dien  the  problem  has  been  solved.  This  formulation  leads 
to  an  interesting  rgtplicatioa  for  chunking:  networks  of  constraints  can  be  satisfied  by  subgoaling  in  such  a  way 
that  chunks  can  be  learned  for  the  efficient  sadsfoedon  of  a  constraint  network,  or  macro-constrauu.  We  have 
demonstrated  this  form  of  constraint  compilation  for  small  networks  of  digital  logic  elements,  with  the  chunks 
being  exaedy  the  productions  that  would  be  built  by  hand  to  compile  the  macio-coastrainL 

Leamiag  from  outside  guidance:  We  have  built  a  system  that  learns  search  control  knowledge  using 
outside  guidance  in  the  domain  of  solving  simple  algebraic  equadons  in  one  unknown  (Golding,  Rosenbloom  A 
Laird.  1987).  The  system  has  a  set  of  algebraic  operators,  such  as  odd  and  commute,  and  must  decide  which  of 
these  operaiors  to  apply  and  how  to  instantiate  them  at  each  step  of  the  soludon.  When  the  system  has 
insufficient  knowledge  to  select  an  opetaior  from  a  set  of  candidates,  it  asks  for  help,  which  can  be  given  to  it  in 
one  of  two  forms:  eidier  it  can  accept  direct  advice,  or  it  can  accept  a  simpler  problem  to  solve  that  will 
iUusoam  the  ptindple  it  is  to  use  in  solving  the  harder  probiem.  If  direct  advice  is  given,  the  system  will  first 
verify  that  the  advice  is  coriect,  thus  proiecting  itself  from  erroneous  advice.  During  the  verification.  Soar 
builds  chalks  oouaininf  the  relevant  search  control  knowledge  for  future  use.  This  particular  type  of  learning 
also  occurs  in  the  teaniog  appreodoe  systems,  such  m  LEAP  (Mitchell,  Mahadevan  A  Steinberg.  1983),  where 
it  is  called  ver^atio»-based  leamiHg  (VBL)  (Mahadevan85. 1983).  If  the  advice  takes  the  form  of  a  simpler 
proUem  to  solve,  Som  atiempa  to  do  so,  either  by  brute-force  search,  or  by  accepting  direct  advice  abom  the 
simpler  proUem.  In  the  process,  Sov  will  build  chunks  that  summarize  the  solution,  which  will  irsnsfor 
directly  to  the  harder  proMem,  provided  that  problem  selected  was  appropriate.  TTiis  is  a  simple  form  of 
analogy;  we  have  not  yet  inveadgatad  more  conqiles  analogies  that  would  attenqn  to  adapt  the  sinqiler  problem 
striudon  to  the  harder  problem  with  deUberate  pracessmg. 
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Abstraction  planning:  Another  kind  of  ptobtem-soiving  behavior  that  has  traditionally  been  obtained  in  AI 
systems  (such  as  ABSTRIPS  (Saceidoti,  1974))  through  special-purpose  mechanisms  is  abstraction  planning. 
Chunking  in  Sov  has  already  been  used  to  achieve  the  e^ts  of  non-abstraction  planning:  ties  in  operator 
selection  impasses  are  often  resolved  by  lookahead,  and  the  chunks  formed  during  the  lookahead  apply  to 
reduce  effort  in  problem-solving  based  on  the  result  of  the  kxAahead.  One  way  to  make  planning  even  more 
cost-effective  is  to  use  abstracted  proUan  spaces  for  the  lookahead  search  (Unruh,  Rosenbloom  &  Laird,  1987). 
Abstraction  of  problem  spaces  in  Soar  is  not  based  on  the  explicit  manipulation  of  a  declarative  specification  of 
operators.  Rather,  the  abstraction  is  a  consequence  of  ignoring  parts  of  the  problem  space  as  needed  duri  ^ 
problem-solving.  The  particular  abstractions  possible  are  determined  by  the  manner  in  which  the  problem  space 
has  been  decomposed  into  independent  subcomponents  such  as  initial  state  creation,  operator  precondition 
testing,  operator  implementation,  state  evaluation  and  goal  testing.  When  searching  in  an  abstracted  problem 
space,  only  the  productions  implementing  the  operations  that  are  part  of  the  abstracted  space  tire,  and  the 
chunks  summarizing  the  processing  in  the  abstracted  search  transfer  to  the  non-abstracted  space.  The  abstracted 
problem-solving  has  paid  attention  to  a  subset  of  what  it  would  normally  examine.  Therefore,  the  chunks 
learned  can  be  more  general  than  those  learned  normally,  and  the  appropriateness  of  the  abstraction  determines 
how  useful  the  chunks  are.  This  was  tested  in  Rl-Soar  with  several  abstractions.  The  abstractions  used  were 
removing  the  backplane-box  operator,  removing  the  modules-backplane  operator,  and  removing  the  state 
information  about  module  width.  Two  abstractions  together  saved  75%  of  decision  cycles,  learning  alone  saved 
60%,  and  learning  together  with  the  two  abstractions  saved  77%.  Issues  currently  being  investigated  in  this 
work  are  the  automatic  creation  and  selection  of  appropriate  abstractions,  and  the  use  of  multiple  levels  of 
abstractions. 

Explanatioa-bascd  generalizatioa;  Using  the  framework  for  EBG  described  by  Mitchell,  Keller  and  Kedar- 
Cabelli  (Mitchell,  Keller,  Kedar-Cabelli,  1986),  we  have  shown  that  chunking  in  Soar  implements  a  variant  of 
explanation-based  generalization  (Rosenbloom  A  Laird,  1986).  The  domain  theory  is  implemented  in  some 
Soar  problem  space,  and  an  explanation  is  the  trace  of  the  problem-solving  in  this  space.  Chunking  uses  this 
trace  in  the  same  way  as  goal  regression  uses  a  proof  or  explanation.  The  aim  of  regression  is  u>  describe  the 
goal  concept  in  terms  of  operational  predicates,  where  chunking  produces  a  description  of  the  concept  in  terms 
of  predicates  on  attributes  of  the  context  that  existed  priw  to  the  impasse.  The  condition  finding  algorithm  in 
Soar’s  chunking  is  essentially  the  same  as  regression  in  EBG  except  it  uses  instantiated  goals  and  rules  rather 
than  parametrized  versions.  The  mapping  woiks  not  only  fev  operator  implementation,  but  also  for  learning 
search  control  knowledge,  where  the  goal  concept  is  the  preference  for  a  selection  of  an  operator.  No  extra 
mechanisms  are  needed  to  produce  the  results  of  EBG  with  chunking;  the  problem  solving  that  is  analogous  to 
EBG's  explanation  is  exactly  the  same  as  is  used  by  Soar  in  other  probtem  solving.  To  test  this  mapping,  we 
took  several  examples  in  (Mitchell,  Keller,  Kedar-Cabelli,  1986)  and  implemented  problem  spaces  in  Soar  that 
acquired  nearly  identical  concepts  to  the  ones  acquired  by  ’’conventional"  EBG  algorithms. 


3  Theoretical  analysis  of  chunking 

Sources  of  ovcrgcaeralisatioo  la  Soar.  Soar  exhibits  overgeneralization  while  chunking  in  several 
instances.  We  now  believe  that  we  understand  most,  if  not  all,  sources  of  overgeneralization  in  Soar  (Laird, 
Rosenbloom  A  Newell,  1986c).  Ow  investigation  was  based  on  the  theory  of  chunking  as  knowtedge 
compUadon,  that  is,  the  conversion  of  knowledge  from  one  forni  to  another  so  that  it  may  be  used  in  similar 
situations  bid  more  efficiently  than  before.  Fbr  this  conversion  to  be  correct  and  avoid  overgeneralization  in 
Soar,  all  of  the  relevant  aspects  of  the  pre-impasse  situation  that  lead  to  the  creation  of  results  must  be  included 
as  conations  of  a  churdt.  This  is  easy  as  long  as  the  results  of  a  subgoal  depend  only  on  the  productions  that 
fired  during  the  wbgoaL  However,  in  some  subgoals,  the  appropriate  result  (such  as  success  or  failure)  is  most 
easily  delected  by  testing  some  feature  of  the  problem  solving  process,  such  as  the  available  operators  are 


Learning  in  Soar:  1987 


8 


exhausted  and  there  nothing  more  to  do.  ^  example,  if  the  highest  number  of  a  set  is  to  be  found,  and  each 
step  of  the  probien  solving  involves  comparing  one  of  the  numben  u>  the  highest  found  so  far,  the  problem  is 
finished  when  no  other  numben  are  left  to  be  compared.  In  Soar,  detecting  thm  there  is  nothing  more  to  do 
means  testing  that  no  moR  productions  can  fire  for  the  current  situation  which  in  Soar  becomes  testing  for  the 
existence  of  an  impasse.  For  chunking  to  be  correct,  it  must  determine  why  no  more  productions  fired  and 
iiKludes  these  reasons  in  the  chunk.  Although  this  may  be  logically  possible,  it  is  computationally  intractable 
and  the  resulting  chunks  can  be  less  efficient  than  the  original  problem  solving.  Therefore,  these  conditions  are 
not  included  and  the  resulting  chunks  are  overgeneral.  This  problem  arises  whenever  the  problem  solving  for  a 
goal  is  dependent  on  its  own  process  state  (meta-information)  which  is  made  available  in  Soar  though  the 
impasses.  Becau.se  of  these  problems,  sve  are  currently  investigaiing  a  variety  of  methods  to  recover  from 
overgeneralization  in  these  cases  when  it  cannot  be  prevented. 

Chooking  and  mcta-ievcis:  We  have  also  analyzed  Soar  in  terms  of  concepts  such  as  meta-levels, 
introspection  and  reflection  (Rosenbloom.  Laird  k  NeweU.  1986).  Several  mem-levels  were  identified,  and 
chunking  was  cast  as  a  means  for  shifting  knowlei;^  from  the  problem-space  meta-level,  where  its  application 
is  slow  and  delibeiaie.  to  the  productioo  meta-level,  where  us  application  is  fast  and  automatic.  This  production 
meta-level  acts  as  a  long-term  cache  for  the  results  of  problem  solving  at  the  higher  problem  q»ce  level,  and 
improves  the  efficiency  of  the  problem  solver  by  reducing  the  need  to  climb  the  hierarchy  of  meta-levels  as 
often. 


4.  Soar  as  a  learning  architecture 

Figure  4-1  lists  some  questions  that  might  be  asked  about  a  particular  teaming  architecture  in  order  vo  assess 
its  abilities  and  compare  it  to  others.  This  section  attempts  u>  answer  these  questions  with  regard  to  Soar. 

•  When  can  the  architecture  leam? 

*  On  what  kinds  of  tasks? 

*  On  what  occasions  during  performance  of  the  tasks? 

•  From  what  can  the  architecture  leam? 

*  Internal  sources  of  knowledge? 

*  External  sources  of  knowledge? 

•  What  can  the  architecture  leam? 

*  What  kinds  of  procedural  knowledge? 

*  What  kinds  of  declarative  knowledge? 

•  When  can  the  architecture  apply  teamed  knowledge? 

*  WtOm  the  same  trial 

*  Acran  different  Dials 

*  Across  different  tasks 

Flgtreed-l:  Questions  u>  ask  about  learning  aichitecuues 


4.1.  When  can  Soar  leant? 

When  worliiBf  ea  difUertat  types  of  tasks:  These  tasks  span  the  range  ftom  simple  and  knowledge-lean  to 
complex  and  knowlet^intensive,  from  routine  to  design  and  discovery.  Soar  has  exhibited  interesting 
teaming  on  instances  of  all  of  these,  and  is  theoretically  capable  of  learning  on  any  task  it  can  perform.  The 
simple  tasks  include  the  eight-puzzle,  dc-tac-toe,  urwers  of  Hanoi,  misskmaries-md-cannibals,  solving  simple 
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algebraic  equatkms  in  one  unknown,  satisfying  logical  constraint  networks,  solving  logical  syllogisms,  balance 
beam,  simile  blocks  world,  eight-queens,  nmnkey-and-bananas,  and  the  water  jug  problems.  The  complex  tasks 
include  computer  configuration,  algorithm  (tesign,  and  ai^Iying  sociological  theories.  Routine  tasks  include 
paired-associate  learning  and  the  Seibel  task.  One  discovery  task  is  algorithm  design. 

Whenever  a  result  is  generated:  The  chunking  mechanism  is  the  same  for  results  at  any  level  in  the  goal 
hierarchy,  and  is  invoked  independently  of  the  success  or  failure  of  the  goal.  For  example,  Rl-Soar  builds  a 
chunk  for  the  top-level  goal  during  the  successful  configuration  of  a  VAX  backplane.  Learning  from  failure 
occurs  during  the  missionaries-and-cannibals  nins;  if  a  move  is  made  that  results  in  there  being  more  cannibals 
than  missionaries  on  a  bank  or  in  the  boat  (an  illegal  state).  Soar  learns  not  to  make  that  move  again. 

4.2.  From  what  can  Soar  learn? 

Internal  sources  of  knowledge:  The  normal  source  of  learning  for  Soar  is  its  own  problem  solving 
experience,  since  it  usually  functions  autonomously.  For  examine.  Soar  has  learned  to  solve  the  missionaries- 
and-cannibals  problems  by  brute-force  search,  and  to  satisfy  macro-constraints  based  on  solutions  to  constraint 
networks  found  previously.  The  mechanism  for  all  this  is  uniform.  The  chunking  mechanism  takes  the  traces 
of  the  problem  solving  as  input.  The  productions  that  led  to  the  creation  of  all  results  are  backtraced.  The 
conditions  of  those  productions  testing  elements  that  existed  prior  to  the  impasse  are  variabilized  and  made  part 
of  the  actions  of  the  chunk.  Any  elements  created  during  the  subgoal  that  are  tied  to  the  superstate  are 
considered  part  of  the  result,  and  corresponding  actions  are  made  part  of  the  right-hand  side  of  the  chunk. 

External  sources  of  knowledge:  Several  of  the  research  efforts  described  earlier  illustrate  how  Soar  can 
learn  from  knowledge  provided  to  it  externally.  In  the  algebraic  domain,  the  user  can  supply  direct  advice  on 
which  of  several  operators  to  apply  to  an  equation.  Soar  can  also  obtain  the  same  knowledge  if  the  user 
supplies  it  with  a  similar,  simpler  problem  to  solve.  The  chunks  built  after  solving  the  simpler  problem  will 
transfer  to  provide  the  search  control  for  the  original  problem.  Two  other  sources  of  external  knowledge  now 
being  t^tped  by  Soar  are  pseudo-natural  language  descriptions  of  task  problem  spaces  and  declarative  facts. 


4.3.  What  can  Soar  learn? 

Procedural  knowledge:  A  step  in  a  procedure  in  Soar  corresponds  to  the  replacement  of  a  single  element  in 
a  goal  context.  A  goal  context  consists  ttf  a  goal,  inoblem  space,  state  and  operator.  We  can  analyze  what 
procedural  knowledge  Soar  can  leam,  and  has  been  demonstrated  to  learn,  by  examining  what  aspects  of  this 
context  replacement  are  subjea  to  modification  by  chunking. 

•  Goals:  The  set  of  possible  goal  types  forms  a  complete  set,  fixed  by  the  architecture:  goal  types  are 
determined  by  the  decisioa  cycle  in  response  to  impasses  in  a  manner  also  fixed  by  the  architecture. 
Qmsequently,  the  effect  of  chunking  on  the  goal  element  of  a  context  is  limit^  to  avoiding  the 
need  for  subgoaling. 

•  Probtea  spaces:  A  problem  space  consists  of  a  set  of  possible  states  and  operators,  and  the 
architecture  does  not  impose  any  restrictions  on  what  constitutes  a  legal  set  Therefore,  Soar  must 
generate  candidate  problem  spaces,  and  select  from  snong  the  candidates.  For  generation.  Soar 
can  either  acquire  problem  spaces  from  external  descriptions,  or  can  modify  problem  spaces  it 
already  has  to  obtain  new  ones.  The  work  on  task  acquisition  has  shown  Soar  to  be  capable  of  the 
former.  For  the  latter,  problem  qtaces  have  been  abstracted  from  old  ones  by  omitting  operators  in 
the  work  on  abstraction  planning,  and  Soar  builds  new  problem  spaces  during  the  baliuice  beam 
task  as  well.  There  is  nothing  preventing  Soar  from  learning  problem  space  selection  knowledge, 
we  have  only  a  few  examples  of  learning  this  knowledge,  in  the  context  of  task  acquisition  and  the 
balance  beam  task. 


•  States:  The  selection  of  a  problem  space  imposes  certain  constraints  on  the  creation  and  selection 
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of  stales.  Our  main  experience  to  dale  with  acquiring  initial  stae  creation  and  selection  knowledge 
has  been  through  task  acquisitioa.  Forcreationofstaiesoiherthaninitialones.  the  processing,  and 
thus  the  knoudedge  to  acquixe  by  chunking,  is  ptimatily  determined  by  the  operator  select^  for 
application  to  the  previous  state.  Much  of  the  minsfer  due  to  chunking  comes  fiom  learning  how  to 
create  states,  or  implement  operators,  without  subgoaling.  FOr  example,  in  Cypress-Soar,  an 
operator  in  the  top-level  space  creates  a  divide-and-conquer  sorting  algorithm  to  satisfy  a  sotting 
^MciHcatioo.  The  first  time  through  the  problem,  this  operator  will  be  impleniented  by  subgoaling 
into  a  special  space  for  designing  divide-and-conquer  algorithms,  and  a  chunk  will  be  built  that 
creates  an  algorithm  such  as  quicksort  for  that  specification.  On  subsequent  trials,  that  chunk  will 
fire  to  create  the  new  state  with  the  algmithm  directly. 

•  Operators:  Operator  creation  is  part  of  task  acquisition,  and  an  equivalent  effect  was  achieved  in 
lewning  macro  tables  for  the  eight-puzzk  and  Tower  of  Hanoi.  Fbr  operator  selection  knowledge, 
examples  abound  in  the  toy  problems,  in  Rl-Soar  and  Cypress-Soar  and  in  the  research  on  learning 
from  outside  guidance  and  abstraction  planning.  Most  opermor  selection,  or  search  control,  chunks 
result  from  summarizing  the  processing  in  lookahead  search,  where  it  is  learned  that  a  particular 
operator  qrplication  will  be  on  the  path  to  a  solution. 

Declarative  knowledge:  The  work  on  recognition,  free  recall,  and  cued  recall  shows  that  Soar  can  learn 
declarative  kmwiedge. 


4.4.  When  can  Soar  apply  learned  knowledge? 

Within  the  same  trial:  Knowledge  that  is  learned  while  solving  a  jHoblem  applied  to  improve  performance 
in  a  lata  stage  of  solving  the  same  problem  is  called  within-trial  tranter.  This  may  be  measured  by  comparing 
the  numba  of  decision  cycles  Soar  takes  to  solve  a  given  problem  without  learning  against  how  many  it  takes 
with  learning.  The  reduction  can  be  substantiaL*  a  factor  of  2  for  some  runs  of  the  eight  puzzle,  ova  a  factor  of 
2  for  Cypress-Soar,  and  a  factor  of  3  for  Rl-Soar.  The  transfer  occurs  because  of  the  presence  of  repeated 
subgoals  during  a  trial  for  these  problems. 

AcroM  different  trials:  When  the  system  builds  chunks  for  every  terminated  goal,  the  only  effort  required 
for  solving  a  problem  on  the  second  and  subsequent  triab  is  the  processing  demand  by  solving  the  problem  with 
no  search.  A  chunk  in  Rl-Sov  reduces  the  numba  of  decision  cycles  in  a  solution  of  the  same  problem  on 
subsequent  trials  by  a  factor  of  20  to  200  in  some  cases.  If  the  system  is  learning  bottom-up,  where  chunks  are 
only  built  for  terminal  goals,  the  processing  that  is  repeated  most  often  gets  chunked  fust  Evennially,  the 
results  of  botiom-up  chunking  will  be  the  same  as  those  of  all-goals  chunking.  Work  on  the  Seibel  reaction¬ 
time  task  has  shown  that  the  across-trial  transfa  with  bottom-up  chunking  leads  to  a  powa  law  effect  on 
performance. 

AcroM  diffcrcBt  tasks:  Across-task  transfa  occurs  in  Soar  when  similw  subgoals  show  up  in  different  tasks; 
this  happened  in  Cypress-Soa,  in  the  Seibd  tatit,  in  several  of  the  toy  problems,  and  to  some  extent  in  Rl-Soa. 
For  example,  in  the  divide-and-oonqua  formulation  of  all  the  sorting  algorithms  synthesized  by  Cypress-Soa,  a 
sub-algorithm  must  be  oeaied  to  sort  lists  of  length  zero  or  one.  Once  this  sub-algorithm,  which  merely  returns 
its  input,  was  <Wtg"«vi  for  one  algorithm,  the  chunk  built  during  that  design  could  fire  on  the  design  of  other 
algorithms.  Once  one  of  the  three  algorithms  had  been  designed,  such  chunks  reduced  by  10  to  20  percent  the 
decision  cycles  needed  lo  design  the  otha  two  algorithms. 
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4  J.  What  variety  of  learning  has  Soar  not  yet  exhibited? 

In  order  to  sunpon  the  claim  that  chunking  is  a  truly  general  learning  mechanism,  it  must  be  studied  further  in 
a  number  of  areas.  A  partial  list  of  these  areas  follows: 

•  Very  large  problems:  We  have  not  yet  tested  learning  in  Soar  in  large  (greater  than  10^  rules) 
systems  to  solving  very  difficult  problems,  though  this  may  soon  be  possible  as  a  consequence  of 
work  in  progress  on  Designer-Soar  (for  algorithm  design  tasks)  and  Neomycin-Soar  (for  medical 
diagnosis  tasks). 

•  Learning  from  exhaustion:  We  are  currendy  investigating  issues  of  learning  from  exhaustion, 
that  is,  learning  when  a  goal  terminates  because  Soar  runs  out  of  things  to  do.  This  is  difficult 
because  the  reason  to  the  exhaustion  is  not  an  explicit  input  to  the  chunl^g  mechanism. 

•  Complex  analogies;  In  learning  from  outside  guidance  provided  as  a  simpler  problem  to  solve. 

Soar  performs  a  rudimentary  form  of  analogy,  which  requires  that  the  chunks  transfer  directly. 

Soar  is  not  yet  capable  of  more  comidex  analogical  reasoning  involving  deliberate  processing  to 
adapt  the  solution  to  the  simpler  problem.  Related  to  this  is  the  problem  of  how  to  use  multiple 
examples. 

•  Representation  shifts:  In  the  category  of  problem  space  creation,  we  have  yet  to  demonstrate  the 
creation  of  "completely  new”  representations  in  a  task.  White  there  is  a  moderate  shift  of 
representation  in  the  balance  beam  task,  it  is  still  the  case  that  the  set  of  possible  problem  ^ace 
modifications  must  be  somehow  "designed  in"  beforehand. 

•  Across-space  transfer:  Soar  has  not  yet  exhibited  transfer  of  knowledge  across  different  problem 
spaces.  Demonstration  of  such  transfer  would  significandy  increase  the  transfer  of  knowledge 
gained  during  the  performance  of  one  task  to  other  tasks. 


5.  Conclusion 

We  have  described  how  Soar  successftiUy  teams  in  a  wide  range  of  tasks  and  situations.  Most  of  these  results 
could  previously  only  be  obtained  from  special-purpose  learning  mechanisms.  The  variety  of  teaming  possible 
in  Soar  does  not  arise  from  collecting  all  these  mechanisms  into  one  system.  Rather,  they  have  been  produced 
with  a  single  mechanism,  which  is  general  enough  to  apply  to  all  problem-solving  behavior.  Thus  the  divenity 
of  problem-solving  of  which  Soar  is  capable  is  directly  responsible  for  the  varieties  of  learning  in  Soar. 
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